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ABSTRACT : 

A computer-based multimedia presentation system is provided with a synchronization 
scheme for recording and playing independent media. The disclosed system and method 
allows media processes and single medium processes to achieve and maintain 
synchronization with each other without process interdependence and without 
interprocess communication. This capability is provided by assigning a common clock 
for all processes, requiring all participating media processes to reference the 
common clock, informing each process of, a synchronization basepoint called a 
"zero-time", and then allowing each process to independently synchronize itself to 
the common clock. The common clock itself does not provide any stimulus to a media 
process; it is a passive component in the synchronization. The media process is the 
active component, referring to the common clock as required to maintain 
synchronization for the particular media it is handling. 

34 Claims, 8 Drawing figures 
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L23: Entry 4 of 4 File: USPT May 30, 1995 



DOCUMENT- IDENTIFIER: US 5420801 A 

TITLE: System and method for synchronization of multimedia streams 
Brief Summary Text (5) : 

One of the important challenges involved in presenting multimedia information in a 
cohesive, life-like fashion is synchronization of the numerous components that make 
up a typical presentation. This synchronization includes two aspects: first, the 
presentation components must start in synchrony; second, they must progress at the 
same rate. Moreover, both aspects of synchronization must be adhered to even when 
the presentation is played after fast -forward or reverse, or is started at a point 
other than its beginning. 

Drawing Description Text (5) : 

FIG. 4 is a timeline illustrating the synchronized playback of multi -media. 
Drawing Description Text (6) : 

FIG. 5 is a timeline illustrating the synchronized playback and recording of 
multimedia. 

Detailed Description Text (2) : 

Shown if FIG. 1 is a high level representation or the system components and 
information flow which provide multimedia synchronization in accordance with the 
present invention. The major components include: starter process 100 which performs 
the functions of determining zero-time at 101 and initiating the media processes at 
103, media processes 104 105 106, also known as media operations, which record 
and/or playback a synchronized multimedia presentation, and clock means 102 which 
provides the current time in the computer system. The current time is the 
interpretation of the temporal bit pattern into standard time nomenclature hours, 
minutes, seconds, milliseconds, etc. 

Detailed Description Text (15) : 

Shown in FIG. 4 is a multimedia playing system in accordance with the present 
invention. Two major components are included in the system. The first is a process 
which establishes zero-time 401 and creates coded zero-time 402. This process is 
called the playback starter process, and is designated as 403. From the coded zero- 
time 302 and the common clock 201 a zero-time , scale 434 can be constructed by any 
process. The second component is one or more processes which use coded zero-time 
402 in conjunction with common clock 201 to play multimedia or single media events. 
These processes are called media playback processes and are designated as 404. 
Examples of media playback processes 404 are audio playback process 405, text 
playback process 406, and image playback process 407. 

Detailed Description Text (16) : 

As common clock 201 is advancing, playback starter process 403 is started to 
establish zero-time 401 during zero-time initialization 408, and initiate media 
playback processes 404 during process initiation 435. Playback starter process 403 
calculates zero-time 401 based upon the current time 422 on common clock 201 and 
the estimated process preparation 409 needed by media playback processes 404. 
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Playback starter process 403 constructs coded zero-time 402 and initiates audio 
playback process 405, text playback process 406, and image playback process 407, 
passing in coded zero-time 402 to each of media playback processes 404. 
Alternatively, playback starter process 403 may retain coded zero-time 402 
available for query by any of media playback processes 404. After media playback 
processes 404 have been initiated, playback starter process 403 can either cease, 
become a member of media playback processes 4 04, or perform other tasks. 

Detailed Description Text (17) : 

Each of media playback processes 404 will perform a process preparation 410 in 
order to prepare all of its internal resources and acquire all of its external 
resources. If zero-time 401 has not passed when one of media playback processes 404 
has completed process preparation 410, that media playback process will perform a 
process wait 411 until zero-time 401 occurs. This time is known as media start- 
time. Each or media playback processes 404 will operate independently or the other 
media playback processes. This is possible because each of media playback processes 
404 is aware of common clock 201 and is provided with coded zero-time 402. 

Detailed Description Text (18) : 

Once audio playback process 405 has been initiated, it begins audio playback 
process preparation 412. Audio playback process 405 acquires control or. the audio 
playback hardware, locates the audio data file(s) (including a log file in which 
the zero- time offsets are given for all audio events, such as, when to start the 
audio, and when to switch audio data files) , prepares storage space for the 
digitized data, and initializes the audio playback hardware, any data buffers 
required for audio playback, as well as locating the common clock, determining the 
first audio event anti and other general initialization If zero-time 401 has not 
passed by the time audio playback process preparation 412 is finished, audio 
playback process 4 05 determines the amount of time it must perform audio playback 
process wait 413 in order to begin the multimedia presentation at zero- time 401. If 
zero-time 401 has passed by the time audio playback process preparation 412 is 
finished, audio playback process 405 advances into the recorded audio data to 
account for the difference in time between zero- time 4 01 and the end of audio 
playback process preparation 412. If the audio data contains information indicating 
that the original recording began after zero-time, audio playback process 405 
continues its audio playback process wait 413 until that amount of time after zero- 
time 4 01. 

Detailed Description Text (19) : 

Once text playback process 406 has been initiated, it begins text playback process 
preparation 414. Likewise, image playback process 407 begins image playback process 
preparation 415. As with audio playback process preparation 412, each process must 
allocate its internal anti external resources. Text playback process 406 then 
determines whether to enter text playback process wait 416 until zero-time 401, 
whether to enter it until after zero-time 401, or whether to advance through the 
text data. Image playback process 407 determines whether to enter image playback 
process wait 417 until zero-time 401, whether to enter it until after zero-time 
4 01, or whether to advance through the image data. 

Detailed Description Text (20) : 

As audio playback process 4 05, text playback process 4 06, and image playback 
process 407 are playing, they periodically check the difference between zero-time 
401 and current time 418. This difference, relative time 419, is used to adjust the 
rate of playing, and to detect the need to skip forward or wait for the next event 
in the case where synchronization is lost. In cases where an unforeseen system load 
has caused the process to fall out of synchronization, the process must adjust the 
rate at which is plays to get back into sync. The process may choose to skip ahead 
if it has fallen behind in the presentation, or it may choose to wait if it has 
gotten ahead. Both skipping and waiting may be implemented using a variety of 
techniques known in the art. Waiting, instance, may be implemented by polling or 
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sleeping. Sleeping is chosen in the preferred embodiment because it allows other 
processes to have the greatest possible access to computer resources. While 
skipping/waiting approach may be appropriate for an image or video process, which 
can simply skip a frame, it is not the best choice for an audio process. Skipping 
forward in audio will result in a discontinuity in the audio. The preferred 
approach^ for audio is to increase the rate at which the audio plays. If the 
adjustment is small this temporary increase in the rate may not be noticeable. 
Similarly, the audio process may need to decrease the rate if the audio process has 
somehow become ahead of time. If large adjustments are needed in the audio rate, 
the frequency spectrum of the audio may be adjusted to preserve the tonal 
properties of the audio, using methods well known in the field of signal 
processing . 

Detailed Description Text (22) : 

Shown in FIG. 5 is a set of time lines depicting the sequence of adding multimedia 
to a presentation. Three processes participate in this sequence. The first process 
establishes zero time at 501 and creates coded zero-time at 502. This process is 
designated as playback/record starter process 503. From the coded zero-time 302 and 
the common clock 201 a zero-time scale 534 can be constructed by any process. The 
second process is in practice one or more processes which use coded zero-time 502 
in conjunction with common clock 201 in playing multimedia or single media events. - 
These processes are called media playback processes, designated generally as 504. 
Examples of media playback processes 504 include audio playback process 505, text 
playback process 506, and image playback process 507. The third process is also 
actually one or more processes, which in this instance use coded zero- time 502 in 
conjunction with common clock 201 in recording multimedia or single media events. 
These processes are called media recording processes 520. An example of a media 
recording process is image recording process 521. 

Detailed Description Text (23) : 

As common clock 2 01 advances , playback/record starter process 503 is started to 
establish zero-time 501 during zero-time initialization 508 and initiate media 
playback processes 504 and media recording processes 520 during process initiation 
535. Playback/record starts process 503 calculates zero-time 501 based upon the 
current time 522 on common clock 2 01 and the estimated process preparation 50 9 
needed by media playback processes 504 and media recording processes 520. 
Playback/ record starter process 503 constructs coded zero-time 502 and initiates 
audio playback process 505, text playback process 506, image playback process 507, 
and image recording process 521, passing coded zero-time 502 to each of media 
playback processes 504 and media recording processes 520. Alternatively, 
playback/ record starter process 503 may keep coded zero-time 502 available for 
query by any of media playback processes 504 or media recording processes 520. 
After media playback processes 504 and media recording processes 520 have been 
initiated, playback/ record starter process 503 can either cease, become a member of 
media playback processes 504, become a member of media recording processes 520, or 
perform other tasks . 

Detailed Description Text (24) : 

During this time, media playback processes 504 and media recording processes 520 
continue to function as described above. The result of the simultaneous playback 
and recording of multimedia is that the newly recorded media will be in 
synchronization with the previously recorded media. Each of the recorded media has 
time synchronization information which can be used by a multimedia player having 
the ability to recognize common clock 201 and coded zero-time 502. 

Detailed Description Text (26) : 

Shown in FIG. 6 is a set of timelines depicting the creation of coded zero-time. 
The process involves two principal components. The first is the current time on 
common clock 201. This is denoted as current time 622. The second is information 
indicative of the anticipated startup times of media recording processes 62 0 and/or 
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media playback processes 604. This information is used to construct estimated 
process preparation time 609. 

Detailed Description Text (33) : 

where E is the offset of the actual event within the presentation 726 729, T is the 
current time 724 730 on the clock 201, Z is the zero-time 701 for the presentation, 
and C is the correction 732 733 needed by the media process 723 in the presentation 
of the media event to reproduce the timing of the actual event. In the case of 
playback, E, T, and Z are given and C is calculated. C, the correction, will result 
in the process waiting to perform the next event, performing the event, or 
advancing within the presentation to "catch up". In the case of record, T, Z, and C 
are given and E is calculated. C, the correction, is 0 since the time of the actual 
event is the same as the time of the recorded media event. Since C=0, the general 
equation becomes: 

Detailed Description Text (38) : 

The above procedure is used to determine the timing for each media event. Where the 
result of the equation is positive, process 723 must wait the number of 
milliseconds given by the result. Where the result of the equation equals zero, 
process 723 plays the event immediately. Where the result of the equation is 
negative, process 723 must "catch up" by the absolute value of the result in 
milliseconds. Each of media playback processes may determine its method for 
regaining synchronization, as described above. 

Detailed Description Text (42) : 

FIG. 8 shows a representation or the use or the present invention in synchronizing 
multimedia during media repositioning commands. Media repositioning commands may 
include fast forward, fast rewind, stop, instant replay, rewind, etc. 

Detailed Description Text (43) : 

The example given in FIG. 8 represents how a user might interact with a system 
which uses zero- time based synchronization and how the system would respond. In 
this example, the user initially wants to see the multimedia presentation from the 
beginning. After watching and listening for a minute or two, the user would like to 
see what's ahead in the multimedia presentation. The user fast forwards twenty-five 
minutes into the presentation and plays from that point for a little over two 
minutes. Realizing that there is some important information earlier in the 
multimedia presentation, the user rewinds back to the twelve minute mark and plays 
to the end of the presentation. 

Detailed Description Text (45) : 

At time 9:47:00.00 on the common clock 201, the user issues command #1 816 "Play" 
which is translated into "Play from 0:00:00" 809. This command is passed to the 
coordination process 802 which performs the actions of the starter process 
described in the "PLAYING MULTIMEDIA" section. It determines zero-time #1 821, 
creates coded zero-time as described in the "CREATING CODED ZERO-TIME" section, 
initiates media processes 806 807 808, and transmits 828 829 830 coded zero-time 
and the offset into the presentation at which to begin, 0:00:00.000 through 
interprocess communication 803 804 805. The media processes 806 807 808 perform the 
actions of the playback processes described in the "PLAYING MULTIMEDIA" section. 
They perform process preparation, potentially perform process wait, and perform 
media events as described in the "USING CODED ZERO-TIME" section. 

Detailed Description Text (46) : 

The user decides after watching/listening to one minute and thirty seconds 824 of 
the presentation that it's time to advance into the presentation. First, the user 
issues command #2 817 "Stop" 810 which is sent to the coordination process 802 and 
transmitted 828 829 830 through interprocess communication 803 804 805 to each of 
the media processes 806 807 808. Each media process 806 807 808 stops presenting 
its media. Second, the user issues command #3-- "Fast Forward" 811. This command is 
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not sent to the media processes 806 807 808; it is handled completely within the 
coordination process 802. The coordination process 802 gives the user a feedback as 
to how far " Fast Forward " has advanced into the multimedia presentation. These two 
commands 810 811 could be combined into one at the user interface level, but would 
be kept as two commands at the operational level. 

Detailed Description Text (47) : 

When the user is notified that the multimedia presentation has been advanced to 
0:25:15, the user issues command #4 818 "Play" which is translated into "Play from 
0:25:15" 812. This command is passed to the coordination process 802 which performs 
the actions of the starter process described in the "PLAYING MULTIMEDIA" section. 
It determines zero- time #2 822, creates coded zero- time as described in the 
"CREATING CODED ZERO-TIME" section, and transmits 828 829 830 coded zero-time and 
the offset into the presentation at which to begin, 0:25:15.000, through 
interprocess communication 803 804 805. The media processes 806 807 808 perform the 
actions of the playback processes described in the "PLAYING MULTIMEDIA" section. 
They perform process preparation, potentially perform process wait, and perform 
media events as described in the "USING CODED ZERO-TIME" section. 

Detailed Description Text (48) : 

The user decides after watching/listening to about two and one-quarter minutes 825 
of the presentation that it's time to return to an earlier spot in the 
presentation. First, the user issues command #5 819 "Stop" 813 which is sent to the 
coordination process 802 and transmitted 828 829 830 through interprocess 
communication 803 804 805 to each of the media processes 806 807 808. Each media 
process 806 807 808 stops presenting its media. Second, the user issues command #6- 
-"Fast Rewind " 814. This command is not sent to the media processes 806 807 808; it 
is handled completely within the coordination process 802. The coordination process 
802 gives the user a feedback as to how far "Fast Rewind " has backed up in the 
multimedia presentation. These two commands 813 814 could be combined into one at 
the user interface level, but would be kept as two commands at the operational 
level . 

Detailed Description Text (49) : 

When the user is notified that the multimedia presentation has backed up to 
0:12:30, the user issues command #7 820 "Play" which is translated into "Play from 
0:12:30" 815. This command is passed to the coordination process 802 which performs 
the actions of the starter process described in the "PLAYING MULTIMEDIA" section. 
It determines zero-time #3 823, creates coded zero-time as described in the 
"CREATING CODED ZERO-TIME" section, and transmits 828 829 830 coded zero-time and 
the offset into the presentation at which to begin, 0:12:30.000, through 
interprocess communication 803 804 805. The media processes 806 807 808 perform the 
actions of the playback processes described in the "PLAYING MULTIMEDIA" section. 
They perform process preparation, potentially perform process wait, and perform 
media events as described in the "USING CODED ZERO-TIME" section. Since the user 
does not issue another command, each media process 806 807 808 continues performing 
82 6 until it has no more events to perform, at which time it behaves as if it were 
sent a "Stop" command. 

Current US Class (2) : 
718 



CLAIMS : 

1. A method for synchronizing initiation of media operations in a multimedia 
recording and playback system, comprising the steps of: 

in a starter process, receiving the current time from a clock means, assigning the 
received current time as an initialization- time value, determining a zero- time 
value by adding a process preparation time to the initialization-time value, 
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broadcasting the zero- time value to at least one media process; and 

in the media process, receiving the zero- time value, upon the current time reaching 
the zero- time value, initiating the media operations, 

thereby, synchronizing initiation of the media operations. 

2. A method for synchronizing initiation as recited in claim 1, wherein the media 
operation is any of image recording, image playback, text recording, text playback, 
audio recording, and audio playback . 

6. A method for synchronizing initiation of media operations in a multimedia 
recording and playback system, comprising the steps of: 

in a starter process, receiving the current time from a clock means, assigning the 
received current time as an initialization-time value, determining a zero-time 
value by adding a process preparation time to the initialization- time value, 
broadcasting the zero- time value to at least one media process; and 

in the media process, receiving the zero-time value, determining a media start-time 
by combining an offset time with the zero-time value, and upon the current time 
reaching the media start -time, initiating the media operation, 

thereby, synchronizing initiation of the media operations. 

8. A method for synchronizing media recording in a multimedia recording and 
playback system, comprising the steps of: 

in a starter process, receiving the current time from a clock means, assigning the 
received current time as an initialization- time value, determining a zero- time 
value by adding a process preparation time to the initialization- time value; 
broadcasting the zero-time value to at least one media process; and 

in the media process, receiving the zero- time value, upon the current time reaching 
the zero-time value, repeating for each media event the sub-steps of obtaining the 
current time from the clock means, determining an event offset based on the zero- 
time value and the current time, logging the event offset in association with the 
media event, recording the media event, 

thereby, synchronizing the media recording. 

18. A method for synchronizing media recording in a multimedia recording and 
playback system, comprising the steps of: 

in a starter process, receiving the current time from a clock means, assigning the 
received current time as an initialization-time value, determining a zero-time 
value by adding a process reparation time to the initialization- time value, 
broadcasting the zero-time value to at least one media process; and 

in the media process, receiving the zero- time value, determining a media start -time 
by adding an offset time to the zero- time value, upon the current time reaching the 
media start- time, repeating for each media event the sub-steps of obtaining the 
current time from the clock means, determining an event offset based on the zero- 
time value and the current time, logging the event offset in association with the 
media event, recording the media event, 

thereby, synchronizing the media recording. 

19. A method for synchronizing media playback in a multimedia recording and 
playback system, comprising the steps of: 
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in a starter process, receiving the current time from a clock means, assigning the. 
received current time as an initialization- time value, determining a zero-time 
value by adding a process preparation time to the initialization- time value; 
broadcasting the zero-time value to at least one media process; and 

in the media process, receiving the zero- time value, accessing a log of event 
offset times associated with a plurality of media events, for each logged event 
offset time, performing the sub-steps of readying the associated media event for 
playing, and when the event offset time arrives, playing the media event, 

thereby, synchronizing the media playback . 

32. A method for synchronizing media playback in a multimedia recording and 
playback system, comprising the steps of : 

in a starter process, receiving the current time from a clock means, assigning the 
received current time as an initialization- time value, determining a zero- time 
value by adding a process preparation time to the initialization- time value; 
broadcasting the zero-time value to at least one media process; and 

in the media process, receiving the zero- time value, accessing a log of event 
offset times associated with a plurality of media events, for each logged event 
offset time, performing the sub-steps of readying the associated media event for 
playing, and when the event offset time arrives, playing the media event, 

thereby, synchronizing the media playback . 
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